Update handling of mxOPAQUE_CLASS objects and Subsystem in MAT_v5.jl #205

foreverallama · 2025-08-20T18:02:13Z

Spinoff from #23 to read the undocumented datatype mxOPAQUE_CLASS. Part of a series of changes to read and write these types across formats v5 and v7.3.

Some context :

Subsystem data is written as a uint8 array. However, this looks like another MAT-file that needs to be converted and read into. This needs to parsed before reading any variables in file.
mxOPAQUE_CLASS variables are written with the following headers - Flags, Variable Name, Type Name, Class Name and Metadata
MATLAB uses the same subsystem format for both v7 and v7.3 files, so starting with v7 is good enough.

Edit:

Added a new file "MAT_subsys.jl" which contains methods for caching, parsing, and retrieving subsystem data to be assigned to an object. With this it should successfully load classdef objects. Additional context regarding how subsystem data is organized below:

MCOS subsystem data is a cell array tagged to a class called "FileWrapper__"
The first cell is a metadata array. It contains 9 blocks of metadata. Most of these blocks are to be interpreted as uint32 integers even though its written in as uint8
-- Block 1 is a version indicator and some offset values
-- Block 2 is a list of class and property names as uint8 integers (null terminated)
-- Block 3 is a list of class IDs
-- Blocks 4 and 6 contain some metadata about how linking property names and property values
-- Block 5 is a list of object ID metadata
-- Block 7 is a list of dynamic properties attached to the object
-- Blocks 8 and 9 are unknown
Cell 2 is empty (probably reserved?)
Cell 3:end-3 are property values (depending on subsystem version it could be up to end-2)
The last 2 or 3 cells are some kind of shared class templates. Only the last cell is known - it contains default property values

foreverallama · 2025-08-21T13:48:03Z

With these changes, full support is added for loading classdef objects in MAT-files in both v5 and HDF5 formats. Classdef objects are returned as a Matrix{Dict{String, Any}}. The Dict is a property name, value dictionary, with an additional key __class__ containing the class name as a String.

The changes support different MAT-file and subsystem versions. It also supports loading all types of MCOS classdef objects (which is most of them), including handle class objects. Some other types I've seen are java and handle (for COM objects) which I don't know how to decode yet, but these are quite rare anyways and probably extremely specific to MATLAB.

Some notes:

I didn't really add a separate test because test/v7.3/struct_table_datetime.mat already seems to contain several objects like datetime string categorical and table. Just updated the test there instead
I just copied the copyright notice template from a different file, but I'm not too sure about adding a copyright notice for MAT_subsys.jl since the code is derived from reverse engineering the file format. Maybe someone else can comment on this?
For most of the classes like datetime or string, you will still need to decode the property map into usable information. It would be good to have some utility functions to do that. I've already documented most of it, I'll get to it some other time though (or maybe someone else may take it up)

Edit:
Squashed and consolidated changes for readability. Adds support for loading mxOPAQUE_CLASS objects from both v7 and v7.3 formats. For a review, the main part of the code would be MAT_subsys.load_subsys! and MAT_subsys.load_mcos_object. I've also highlighted some parts I'm not sure about with FIXME or TODO

* MAT_subsys.jl: New file MAT_subsys with methods to set, parse and retrieve subsystem data * MAT_v5.jl: New method "read_opaque" to handle mxOPAQUE_CLASS * MAT_v5.jl: New method "read_subsystem" to handle subsystem data * MAT.jl (matread): Update to clear subsystem and object cache after load Support for loading mxOPAQUE_CLASS objects in v7.3 HDF5 format * MAT_HDF5.jl (matopen): New argument Endian indicator, Reads and parses subsystem on load * MAT_HDF5.jl (close): Update to write endian header based on system endianness * MAT_HDF5.jl (m_read::HDF5.Dataset): Update to handle MATLAB_object_decode (mxOPAQUE_CLASS) types * MAT_HDF5.jl (m_read::HDF5.Group): Update to read subsystem data and function_handles * MAT.jl (matopen): Update function calls Updated test for struct_table_datetime.mat to ensure accurate deserialization (including nested properties) in both v7 and v7.3 formats * test/read.jl: Update tests for "function_handles.mat" and "struct_table_datetime.mat"

…n offests

matthijscox · 2025-11-11T09:37:15Z

Hi there, I like your PR and I hope you still want to work on it.

I have my own PR-207 and I will use Array{Dict{String,Any}} to write struct arrays, and maybe in the future we can also read struct arrays like that. I was wondering if there will be a conflict with your PR, since you will read (not write?) MATLAB classes as Array{Dict{String,Any}}.

Is there a good way to separate normal struct arrays from MATLAB classes in the MAT.jl interface? Would the distinction come from the existence of the "__class__" key inside each Dict in the Array{Dict{String,Any}}?

(I also hope an active contributor/maintainer pops up for MAT.jl to review our work. I am trying to get in contact with them via email)

foreverallama · 2025-11-11T11:29:01Z

Hey!

It does look like there will be a conflict in our PRs. I aimed to provide read-write support in this module, but I only included read support for MATLAB classes in this PR to make it reasonable to review.

Coming to the typing, my opinion is that the same types should be used for both read and write, and the typing should be consistent across the different MAT-file versions. I worked on a Pythonic version of this module, and there I used a wrapper class MatioOpaque to identify MATLAB classes instead. Something like that could be done here as well, or as you mentioned the __class__ key (or some other special key) could be used as an identifier.

Ultimately, it's a design choice that would benefit from a discussion surrounding user requirements. It would be good if a maintainer (@ViralBShah ?) could pitch in their thoughts. As I understand there's not much activity here, but I would be happy to help out in any way I can!

matthijscox · 2025-11-11T14:53:24Z

I like the idea of a wrapper type, at least for read/write consistency. Having obscure key names like __class__ inside a Dict seems a little dodgy.

I wonder if it would be smart to have another wrapper type for the struct arrays? Some kind of mutable named tuple like type would be great, like: Array{MatlabStruct{(:field1, :field2)}}. And/or maybe actually use StructArrays.jl

Looking at matio for inspiration, I found a struct array test, but I cannot quickly infer what Python type you are reading/writing? How do you differentiate between cell arrays of structs and struct arrays? Right now MAT.jl does not have read/write consistency for struct arrays (they get re-written as a struct with cell arrays per field name).

ViralBShah · 2025-11-11T16:12:58Z

@matthijscox is a maintainer and we can add others too.

foreverallama · 2025-11-12T08:33:07Z

How do you differentiate between cell arrays of structs and struct arrays?

So my main point was to keep it consistent with scipy which uses numpy arrays to represent MATLAB types. Struct and Cells are both numpy arrays. The only difference is structs use numpy record arrays with each field having an object dtype, whereas Cell arrays have dtype=object.

For example,

struct_arr_dtype = [('field1', object), ('field2', object)]
cell_arr_dtype = object

arr = some_arr
if arr.dtype.hasobject:
    if arr.dtype.names:
        # struct array
    else:
        # cell array

The only place this doesn't work is for empty structs which don't have field names. In this case, I've used a wrapper class EmptyMatStruct that contains the shape of the empty struct.

Having obscure key names like class inside a Dict seems a little dodgy

I do agree with this. In fact, I went through several iterations in my Python module before arriving at the current data representations. The main problem I had was ensuring separate types for simple read-write operations whilst keeping it user-friendly and scalable for the newer MATLAB classdef-based types. This lead to me using a bunch of wrapper classes that keep a balance between allowing numpy array operations and differentiability.

Something of that sort can be done here if it aligns with the scope of this package, and I would be happy to help. I'm not familiar with Julia typing and will need some time, but in the meantime you can read the type conversions I used in matio, relevant code here. I'll also take a look at MAT.jl and see how we can unify the schema. Once a data representation is fixed it should be pretty straightforward from there.

matthijscox · 2025-11-13T07:50:59Z

src/MAT_subsys.jl

+
+    object_arr = Array{Dict{String,Any}}(undef, convert(Vector{Int}, dims)...)
+
+    for i = 1:length(object_arr)


minor issue: more Julian syntax would be for (i, oid) in enumerate(object_ids)

matthijscox · 2025-11-13T07:55:07Z

src/MAT_subsys.jl

+end
+
+const subsys_cache = Ref{Union{Nothing,Subsys}}(nothing)
+const object_cache = Ref{Union{Nothing, Dict{UInt32, Dict{String,Any}}}}(nothing)


The code is pretty complicated, so I cannot oversee the decisions here, but I am curious why you need this stateful cache? Is it not possible to rewrite the code to just pass the subsys_cache and object_cache through each function?

I think we now we have the risk that someone tries to read multiple .mat files in multiple threads and this will break down.

It's been a while so I'm not sure exactly, but I think it had to with it involving a whole bunch of function definition changes to incorporate both the caches. There's a lot of recursive calls so the caches would have to be passed to all functions even if they don't actually use it.

I would prefer a better alternative to handle thread safety, but your suggestion works as well.

matthijscox · 2025-11-13T08:25:25Z

I've begun reading the code. The classes seem to be read mostly here as dicts in MAT_subsys.jl R285-R288.

I suppose we could define a MatlabOpaque (or MatlabClass or MatlabObject) type that mirrors a dict:

struct MatlabOpaque{T}
    classname::Symbol
    properties::Vector{String} # or Vector{Symbol}
    values::Vector{T}
end

Optionally I wonder if we want to be able to expose the class "type" to the Julia type system. (These Val types do come with a small performance penalty I remember)

struct MatlabOpaque{C<:Val, T}
    properties::Vector{String} # or Vector{Symbol}
    values::Vector{T}
end
# outer constructor
function MatlabOpaque(classname::Symbol, properties::Vector{String}, values::Vector{T}) where T
   return MatlabOpaque{Val{classname}, T}(properties, values)
end

Now you can identify a vector of same class (class/struct array?) from a vector different classes (cell array?)

MatlabOpaque{Val{:table}, Any}[] # class array of Matlab tables
MatlabOpaque[] # class (cell) array of different classes
Any[] # cell array of any kind of (matlab) type

This also means we can write automatic converters:

Base.convert(::Type{DataFrame}, MatlabOpaque{Val{:table}}) = ...

or better yet add the Tables.jl interface:

Tables.istable(::Type{MatlabOpaque{Val{:table}}) = true
Tables.istable(::Type{MatlabOpaque}) = false

Ofcourse instead of the Val type parameter, we could just convert immediately to different known types during the reading. Then we have to define the multiple MAT.jl internal types we need, or immediately convert to the most sensible Julia type.

struct MatlabTable 
...
end
Tables.istable(::Type{MatlabTable}) = true

struct MatlabDuration <: Dates.AbstractTime # or use Dates.Millisecond type
...
end

I would propose to first implement the MatlabOpaque type (with or without Val parameter) in this PR and then in future PRs we can think about the (automatic) conversion to Julia types for table, datetime, duration, etc.

foreverallama · 2025-11-13T14:03:37Z

If I'm understanding correctly, we'd have MatlabOpaque{Val{:classname}, Any}[] for MATLAB object arrays. So this resolves the conflict with struct arrays you proposed?

Some points to clarify/get clarified:

The MatlabOpaque type would have two vectors for properties and values. Do we retain dictionary behaviour instead of indexing? Idk about the performance point of view, but dict behaviour would be helpful, especially for conversions to Julia types as MATLAB types can contain like 10-20 properties for complex types like timetable.
Can we currently load/represent 1x0 0x1 or 0x0 structs with the Dict type? We need that functionality to write classdef objects to MAT-files.
I guess object arrays would then be Matrix{MatlabOpaque{T}}(undef, dims) right?
What do you suggest for mxOBJECT_CLASS and mx_FUNCTION_CLASS. The serialized data for these types are saved as structs, and MAT.jl currently loads them as Dict with an extra key class.

The rest sounds good. Doesn't sound like it diverges too much from the current state of the PR. To summarize the main edits:

Thread safe cache handling
Change return type to MatlabOpaque

matthijscox · 2025-11-13T14:45:12Z

If I'm understanding correctly, we'd have MatlabOpaque{Val{:classname}, Any}[] for MATLAB object arrays. So this resolves the conflict with struct arrays you proposed?

Indeed.

Note I now created a MatlabStructArray type in my PR to fully avoid any conflict with your types I hope.

The MatlabOpaque type would have two vectors for properties and values. Do we retain dictionary behaviour instead of indexing? Idk about the performance point of view, but dict behaviour would be helpful, especially for conversions to Julia types as MATLAB types can contain like 10-20 properties for complex types like timetable.

You mean that you want to index by name object["name"] or object[:name]? Coincidentally I added Dict-like behavior to my MatlabStructArray type (string-based indexing and iteration returns key-value pairs). You can have a look for inspiration: see here for example.

Basically we can define methods like:

function Base.getindex(object::MatlabOpaque, prop::AbstractString)
   idx = findfirst(isequal(prop), object.properties)
   return object.values[idx]
end

Can we currently load/represent 1x0 0x1 or 0x0 structs with the Dict type? We need that functionality to write classdef objects to MAT-files.

Pff, I will have to check. You also make me worried about these empty sized struct arrays for my PR.

Is this the way to construct them?

>> f = {'a', 'b', 'c'};
>> f{2,1} = {};
>> s = struct(f{:})

ans = 

  0×0 empty struct array with fields:

    a
    b
    c

>> reshape(s, 0, 1);
>> reshape(s, 1, 0);

I guess object arrays would then be Matrix{MatlabOpaque{T}}(undef, dims) right?

Yeah, well Matrix{MatlabOpaque{C,T}}(undef, dims) if we choose the Val-based dispatching option. Though I'm thinking now that we can probably drop the T parameter entirely in the struct definition if it's always of type Any. Which seems likely since we don't often expect the same types between MATLAB properties?

What do you suggest for mxOBJECT_CLASS and mx_FUNCTION_CLASS. The serialized data for these types are saved as structs, and MAT.jl currently loads them as Dict with an extra key class.

I very new to MAT.jl myself I admit. I saw the mxOBJECT_CLASS in the MAT_v5.jl file and I'm not sure what it is? Is it the old matlab object class created in @classname folders? I don't see it mentioned in MAT_HDF5.jl (v7.3).

It's possible we're going to have merge conflicts with my PR, since I've been refactoring a bit there. So if you have any time to review my PR that would be great.

matthijscox · 2025-11-13T15:00:00Z

Can we currently load/represent 1x0 0x1 or 0x0 structs with the Dict type? We need that functionality to write classdef objects to MAT-files.

To quickly follow-up on your question:

The bad news

Currently (MAT v0.10) we cannot. An empty 0x0 struct array will be read as Dict{String, Any}("c"=>Matrix{Any}(undef, 0, 0), "b"=>Matrix{Any}(undef, 0, 0), "a"=>Matrix{Any}(undef, 0, 0)) and written back as a 1x1 struct with empty cell arrays in the field:

>> s00

s00 = 

  struct with fields:

    c: {}
    b: {}
    a: {}

Good news

In my new struct array PR I can write them! (reading goes wrong, I'll fix that):

In Julia:

empty_sarr = MAT.MatlabStructArray(["a", "b", "c"], [Matrix{Any}(undef, 0, 0), Matrix{Any}(undef, 0, 0), Matrix{Any}(undef, 0, 0)])
matwrite("test.mat", Dict("s00" => empty_sarr))

In MATLAB:

>> load('test.mat')
>> s00

s00 = 

  0×0 empty struct array with fields:

    a
    b
    c

foreverallama · 2025-11-13T15:17:56Z

You also make me worried about these empty sized struct arrays for my PR

Haha yeah, sorry. It's an edge case from a user perspective, but MATLAB uses these zero-element dimensions as placeholders, and getting them wrong results in unexpected errors when loading in MATLAB. Ideally if the read_struct and write_struct methods can take care of this I can simply use MatlabStructArray in subsystem. In fact it might be better to finish your PR first and then come back to this.

we can probably drop the T parameter entirely in the struct definition

yeah, sounds good.

I saw the mxOBJECT_CLASS in the MAT_v5.jl file and I'm not sure what it is? Is it the old matlab object class created in @classname folders? I don't see it mentioned in MAT_HDF5.jl (v7.3).

I believe mxOBJECT_CLASS is legacy classdef implementation in MATLAB. I guess we can keep it as is, maybe add MatlabFunction and MatlabObject types later if required.

I did notice some differences between the HDF5 and v5 versions. The HDF5 file isn't as extensive as the v5 file, and I honestly think it needs a rewrite to allow consistency between versions. But that's for another time.

matthijscox · 2025-11-14T13:26:03Z

I now support empty struct arrays in my PR #207. Create one easily via MatlabStructArray(["a", "b"], (0,0))

I also created the MatlabClassObject to better support the mxClassObject in v5 format and read-write the class object in v7.3 HDF5 format.
I allow class object arrays via MatlabStructArray if a class name is provided.

I think that's all we need to continue with your PR. I will probably merge soon and then if you want I can help with this PR to solve any merge conflicts for you.

foreverallama · 2025-11-17T07:44:00Z

Thanks! I'm a bit swamped till the end of the month and will most probably take a look at this after, will try to squeeze something in as soon as I can

foreverallama · 2025-11-18T14:53:06Z

src/MAT_subsys.jl

+        merge!(prop_dict, get_properties(subsys, oid))
+        # cache it
+        obj = MatlabOpaque(prop_dict, classname)
+        subsys.object_cache[oid] = obj


The object should be cached before merge!. Else get_properties can result in an infinite recursion for handle class objects

I see... Should we add a unit test for these handle class objects?

Yes that'll be a good idea, I'll add tests for handle class objects and object arrays.

foreverallama · 2025-11-18T14:56:52Z

src/MAT_subsys.jl

+
+function get_default_properties(subsys::Subsystem, class_id::UInt32)
+    prop_vals_class = subsys.prop_vals_defaults[class_id+1, 1]
+    # is it always a MatlabStructArray?


I don't think prop_vals_class will be a MatlabStructArray. It should always be a scalar 1x1 struct. Did you come across an example that goes against this?

Also, we can get rid of the copy statement. Instead, something like this to identify nested objects should work.

for (k, v) in prop_vals_class prop_vals_class[k] = update_nested_props!(v, subsys) end

I had errors for 0x0 MatlabStructArray so this case definitely happens in the test data somewhere

Okay makes sense, MATLAB writes 0x0 structs if there are no default values. But we'll still need to loop through the values looking for nested objects. I'll add a test for nested objects in saved and default values as well.

foreverallama · 2025-11-18T15:01:11Z

src/MAT_types.jl

+            return DateTime[]
+        end
+        if !isempty(obj["tz"])
+            @warn "no timezone conversion yet for datetime objects. timezone ignored"


might be a good idea to display the timezone the datetime object was saved with in the warning?

Sure. Note I didn't want to add TimeZones.jl yet because I remember that it's a massive dependency, adding lots of precompile or artifact download time

foreverallama · 2025-11-18T15:08:47Z

src/MAT_types.jl

+        if num_strings==1
+            return first(strings)
+        else
+            return reshape(strings, shape...)


looks like the job is failing here. Maybe explicitly cast to a tuple?

matthijscox · 2025-11-18T17:22:06Z

Somehow I increased SnoopCompile invalidations a bit, this might mean package loading is slightly slower, but it's all due to dependencies (mostly HDF5.jl). Not sure I know how to fix this, so maybe we'll just take the hit.

On master branch, according to CI:

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────┬───────────────┬─────────────────┐
│ <file name>:<line number>                                                                                                            │ Function Name │ Invalidations │ Invalidations % │
│                                                                                                                                      │               │               │     (xᵢ/∑x)     │
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────┼───────────────┼─────────────────┤
│ HDF5/Z859u/src/dataspaces.jl:204                                                                                                     │    convert    │      830      │      62.44      │
│ /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/SparseArrays/src/readonly.jl:23          │   eachindex   │      146      │     10.984      │
│ HDF5/Z859u/src/datasets.jl:332                                                                                                       │     write     │      146      │     10.984      │
│ /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/SparseArrays/src/solvers/spqr.jl:143     │     axes      │      51       │      3.838      │
│ /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/SparseArrays/src/sparsevector.jl:1667    │  _mapreduce   │      36       │      2.709      │
│ /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/SparseArrays/src/solvers/cholmod.jl:1354 │     size      │      30       │      2.258      │
│ HDF5/Z859u/src/filters/filters.jl:428                                                                                                │    convert    │      28       │      2.107      │
│ HDF5/Z859u/src/datasets.jl:703                                                                                                       │  IndexStyle   │      25       │      1.881      │
│ BufferedStreams/YvcPf/src/emptystream.jl:81                                                                                          │     write     │      20       │      1.505      │
│ BufferedStreams/YvcPf/src/bufferedinputstream.jl:152                                                                                 │     peek      │       6       │     0.4514      │
│ TranscodingStreams/O3BYF/src/stream.jl:245                                                                                           │     reset     │       3       │     0.2257      │
│ TranscodingStreams/O3BYF/src/stream.jl:229                                                                                           │   ismarked    │       2       │     0.1505      │
│ TranscodingStreams/O3BYF/src/stream.jl:234                                                                                           │     mark      │       2       │     0.1505      │
│ /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/LibGit2/src/types.jl:1359                │     get!      │       2       │     0.1505      │
│ /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/SparseArrays/src/sparsevector.jl:2292    │     sort!     │       1       │     0.07526     │
│ /cache/build/tester-amdci4-14/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/LibGit2/src/walker.jl:77                 │     sort!     │       1       │     0.07526     │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────┴───────────────┴─────────────────┘

On my current local commit:

┌──────────────────────────────────────────────────────────────────────────────────────────────┬───────────────┬───────────────┬─────────────────┐
│ <file name>:<line number>                                                                    │ Function Name │ Invalidations │ Invalidations % │
│                                                                                              │               │               │     (xᵢ/∑x)     │
├──────────────────────────────────────────────────────────────────────────────────────────────┼───────────────┼───────────────┼─────────────────┤
│ C:\\Users\\matcox\\.julia\\packages\\HDF5\\Z859u\\src\\dataspaces.jl:204                     │    convert    │      581      │      28.84      │
│ C:\\workdir\\usr\\share\\julia\\stdlib\\v1.12\\SparseArrays\\src\\readonly.jl:23             │   eachindex   │      418      │      20.75      │
│ C:\\Users\\matcox\\.julia\\packages\\StringEncodings\\unzPp\\src\\encodings.jl:17            │    convert    │      357      │      17.72      │
│ C:\\Users\\matcox\\.julia\\packages\\PooledArrays\\Vy2X0\\src\\PooledArrays.jl:499           │    convert    │      286      │      14.2       │
│ C:\\Users\\matcox\\.julia\\packages\\HDF5\\Z859u\\src\\attributes.jl:31                      │     write     │      92       │      4.566      │
│ C:\\Users\\matcox\\.julia\\packages\\HDF5\\Z859u\\src\\datasets.jl:703                       │  IndexStyle   │      75       │      3.725      │
│ C:\\workdir\\usr\\share\\julia\\stdlib\\v1.12\\SparseArrays\\src\\solvers\\spqr.jl:143       │     axes      │      51       │      2.533      │
│ C:\\workdir\\usr\\share\\julia\\stdlib\\v1.12\\SparseArrays\\src\\sparsevector.jl:1667       │  _mapreduce   │      38       │      1.887      │
│ C:\\workdir\\usr\\share\\julia\\stdlib\\v1.12\\SparseArrays\\src\\solvers\\cholmod.jl:1354   │     size      │      30       │      1.489      │
│ C:\\Users\\matcox\\.julia\\packages\\HDF5\\Z859u\\src\\filters\\filters.jl:428               │    convert    │      28       │      1.391      │
│ C:\\Users\\matcox\\.julia\\packages\\PooledArrays\\Vy2X0\\src\\PooledArrays.jl:513           │    convert    │      15       │     0.7446      │
│ C:\\workdir\\usr\\share\\julia\\stdlib\\v1.12\\SparseArrays\\src\\higherorderfns.jl:1188     │     map!      │      14       │     0.6953      │
│ C:\\Users\\matcox\\.julia\\packages\\BufferedStreams\\YvcPf\\src\\emptystream.jl:81          │     write     │      12       │     0.5957      │
│ C:\\Users\\matcox\\.julia\\packages\\BufferedStreams\\YvcPf\\src\\bufferedinputstream.jl:152 │     peek      │       6       │     0.2979      │
│ C:\\Users\\matcox\\.julia\\packages\\TranscodingStreams\\O3BYF\\src\\stream.jl:229           │   ismarked    │       4       │     0.1986      │
│ C:\\Users\\matcox\\.julia\\packages\\TranscodingStreams\\O3BYF\\src\\stream.jl:234           │     mark      │       4       │     0.1986      │
│ C:\\Localdata\\Repositories\\MAT.jl\\src\\MAT.jl:244                                         │     names     │       2       │     0.0993      │
│ C:\\workdir\\usr\\share\\julia\\stdlib\\v1.12\\SparseArrays\\src\\sparsevector.jl:2292       │     sort!     │       1       │     0.04965     │
└──────────────────────────────────────────────────────────────────────────────────────────────┴───────────────┴───────────────┴─────────────────┘

Seems I managed to suppress a few HDF5 invalidations (probably by defining a few more explicit typed m_write functions), but I gained a bunch of SparseArrays invalidations (confused about that one) and some for the new dependencies (StringEncodings and PooledArrays).

matthijscox · 2025-11-19T10:17:29Z

Slightly confused right now. Locally on Julia 1.12.0 I have 883 invalidations. CI (on 1.12.1) says 1474 invalidations

Switched to Julia 1.12.1 with a fresh environment. Got 1040 invalidations

foreverallama · 2025-11-19T17:09:28Z

I added tests for handle class objects and nested properties. I also added support for loading dynamic properties (with tests). Made some tiny changes to accommodate - object arrays are now Array{MatlabOpaque} instead of Array{Any}.

I think we should clean up this PR and focus on supporting more classes/types in a different one. I will add write support for MatlabOpaque after this PR is merged.

matthijscox · 2025-11-19T18:12:00Z

I agree. Do you see any more cleanup needed? Because I'd love to release MAT.jl v0.11 after merging this PR.

matthijscox · 2025-11-19T18:41:02Z

object arrays are now Array{MatlabOpaque}

I think this will fail if one the objects in the array gets automatically converted, e.g. to a DateTime. I will add such a test case tomorrow.

foreverallama · 2025-11-20T07:29:08Z

Do you see any more cleanup needed?

I'll just add some comments to try and improve readability of MAT_subsys.jl. I might be a bit biased though and could use another perspective for suggestions. I updated workflows in matio after initially drafting this PR, I'll just make sure everything's aligned well. Any other linting issues can be corrected now. Good time to update docs and readme as well.

The invalidations stuff is pretty new to me. If I understand correctly, the number of invalidations in the PR must be less than that in master? If its not critical, can we bypass it and tackle it in a separate issue?

I think this will fail if one the objects in the array gets automatically converted, e.g. to a DateTime

Ah so this is a unique quirk of MATLAB, and shouldn't be an issue. For user-defined classes, object arrays contain a separate object instance in each element of the array (unless they are handle class objects which may be references). Also, all objects in an array must be of the same class. But for MATLAB datatypes like datetime and others, it is always a scalar even if the underlying data is an array. MATLAB overloads assignment operations in the class definition, ensuring that only one object instance is created and that its corresponding properties store the array behaviour instead.

matthijscox · 2025-11-20T08:03:46Z

The invalidations stuff is pretty new to me. If I understand correctly, the number of invalidations in the PR must be less than that in master? If its not critical, can we bypass it and tackle it in a separate issue?

Yeah I can merge with the "invalidations CI step" failing. Then I suppose we have a new baseline.

Ah so this is a unique quirk of MATLAB, and shouldn't be an issue. For user-defined classes, object arrays contain a separate object instance in each element of the array (unless they are handle class objects which may be references). Also, all objects in an array must be of the same class. But for MATLAB datatypes like datetime and others, it is always a scalar even if the underlying data is an array. MATLAB overloads assignment operations in the class definition, ensuring that only one object instance is created and that its corresponding properties store the array behaviour instead.

If this is true, that means we can actually skip the convert_opaque call in the for loop for Array{MatlabOpaque}

Any other linting issues can be corrected now.

We could add a formatting style to MAT.jl, none has been chosen so far. I typically opt for bluestyle. We can at least reformat our new files in this PR.

I'll just add some comments to try and improve readability of MAT_subsys.jl. I might be a bit biased though and could use another perspective for suggestions.

The MCOS system seems obscure and the code obvious reflects that a bit. I'm just happy it works at all. We can refactor and comment on the obscure parts, but I'm okay with what we have. (Though what the on earth are some of these fields like _u6_metadata even doing)

…names!

matthijscox · 2025-11-20T08:28:00Z

I took the liberty to refactor a bit and format the style for MAT_types.jl and MAT_subsys.jl

foreverallama · 2025-11-20T08:44:13Z

Sounds good. Everything with subsystem is reverse engineered so we can only guess as to what they do. Some fields are unknown still, but we still need them to write back to a MAT-file. Interestingly, MATLAB does not perform any sanity checks when loading subsystem data, so you can easily crash it by simply going out of bounds in array indexing. For safety, we could add sanity checks here if required.

I wanted one thing clarified before we move forward. I noticed that both MATLAB char arrays and string arrays are returned as Julia strings. It would be better if we could somehow separate this behaviour. Char arrays are written as a bunch of ASCII or UTF-8 encodings, but string arrays use the MCOS opaque class format. It's not an issue in read, but it might be an issue when trying to differentiate during write operations. One of the issues I've come across earlier is that some properties in datatypes like timetable expect MATLAB strings, and will not work with char arrays in MATLAB. I resolved this issue in matio by using NumPy's unicode strings for char arrays (same as SciPy) and using the newer StringDType() for MATLAB strings.

What are your thoughts on this?

Finally, I'm listing a bunch of issues we can close after merging this PR.
#201 #194 #166 #158 #133 #127 #43
#82 can soon be closed as well
And finally we can close the other PR opened a long time back in #23 !

matthijscox · 2025-11-20T09:57:59Z

One of the issues I've come across earlier is that some properties in datatypes like timetable expect MATLAB strings, and will not work with char arrays in MATLAB. I resolved this issue in matio by using NumPy's unicode strings for char arrays (same as SciPy) and using the newer StringDType() for MATLAB strings.

I see. We can obviously postpone until the writing PR. We could use the same approach as you do, but then we need to search for another Julia string type other than the base UTF-8 String. Perhaps one of the LegacyStrings types? And then we have to decide which of the two (char arrays or string classes) is used for the base String. I suppose char arrays are UTF-8?

matthijscox · 2025-11-20T10:09:51Z

Seems to me this PR is ready. Let me know if you are happy with the code or want to do further refactoring. Once you give the go, I'll merge and register v0.11.

…property types

foreverallama · 2025-11-20T11:22:47Z

Lets go for it. We can open new issues for the other stuff, they won't affect base functionality.

I suppose char arrays are UTF-8?

I think they're a mix of ASCII or UTF-8, even though MATLAB documentation says they use UTF-16. It might be optimized during save based on content? MATLAB strings though are definitely using UTF-16.

foreverallama mentioned this pull request Aug 20, 2025

WIP: Read undocumented datatypes #23

Closed

foreverallama force-pushed the subsys branch 2 times, most recently from 9f44ce5 to 89572a7 Compare August 20, 2025 23:58

foreverallama force-pushed the subsys branch from a695f43 to bea378a Compare August 21, 2025 14:30

MAT_subsys.load_subsys!: Fix indexing for parsing metadata from regio…

33c6455

…n offests

matthijscox reviewed Nov 13, 2025

View reviewed changes

matthijscox mentioned this pull request Nov 14, 2025

duration support #201

Closed

matthijscox added 8 commits November 17, 2025 10:32

Merge branch 'master' into pr/205

2bf9c3a

first step in subsystem refactoring

02444a6

stateless MAT_subsys module

c2b55cd

test read(matopen(filepath), ::String) for opaque subsystems

2e6b848

MatlabOpaque to_string conversion

3090b7f

MatlabOpaque to_datetime conversion

e0c9055

fix to_string test

bac5677

MatlabOpaque to_duration conversion

fbde008

foreverallama commented Nov 18, 2025

View reviewed changes

MatlabOpaque table support

edefb99

matthijscox added 5 commits November 19, 2025 09:28

Support for single row tables and ND columns

386f67f

implement PR comments

482d10a

document type conversions

4469b40

small refactor

c94581f

reduce SnoopCompile invalidations

513be33

foreverallama added 2 commits November 19, 2025 22:02

Tests for nested objects, handle class objects

871662e

Add support for loading dynamic properties

8455021

matthijscox added 2 commits November 20, 2025 09:14

mcos refactor: avoid opaque array conversion and extracted load_mcos_…

44b0dc4

…names!

bluestyle formatting

ae064f0

avoid printing massive objects

e1f2f96

MAT_subsys.jl: Display warning instead of throwing error for unknown …

7f3e153

…property types

matthijscox merged commit 45f8231 into JuliaIO:master Nov 20, 2025
7 of 8 checks passed

MegaShark1911 mentioned this pull request Nov 21, 2025

Writing of (compressed) v5 MAT file with MATLAB Class Object Subsytem (MCOS) tbeu/matio#47

Open


		object_arr = Array{Dict{String,Any}}(undef, convert(Vector{Int}, dims)...)

		for i = 1:length(object_arr)

Update handling of mxOPAQUE_CLASS objects and Subsystem in MAT_v5.jl #205

Update handling of mxOPAQUE_CLASS objects and Subsystem in MAT_v5.jl #205

Uh oh!

Conversation

foreverallama commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Edit:

Uh oh!

foreverallama commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matthijscox commented Nov 11, 2025

Uh oh!

foreverallama commented Nov 11, 2025

Uh oh!

matthijscox commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ViralBShah commented Nov 11, 2025

Uh oh!

foreverallama commented Nov 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthijscox commented Nov 13, 2025

Uh oh!

foreverallama commented Nov 13, 2025

Uh oh!

matthijscox commented Nov 13, 2025

Uh oh!

matthijscox commented Nov 13, 2025

The bad news

Good news

Uh oh!

foreverallama commented Nov 13, 2025

Uh oh!

matthijscox commented Nov 14, 2025

Uh oh!

foreverallama commented Nov 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthijscox commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matthijscox commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

foreverallama commented Nov 19, 2025

Uh oh!

matthijscox commented Nov 19, 2025

Uh oh!

matthijscox commented Nov 19, 2025

Uh oh!

foreverallama commented Nov 20, 2025

Uh oh!

matthijscox commented Nov 20, 2025

Uh oh!

matthijscox commented Nov 20, 2025

Uh oh!

foreverallama commented Aug 20, 2025 •

edited

Loading

foreverallama commented Aug 21, 2025 •

edited

Loading

matthijscox commented Nov 11, 2025 •

edited

Loading

matthijscox commented Nov 18, 2025 •

edited

Loading

matthijscox commented Nov 19, 2025 •

edited

Loading